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Abstract 

This research seeks to contribute to current discussions in Australian higher education on how best to 
deploy ICT-enabled learning. Its particular focus is on examining the qualitative data from students on 
their experience of using Information and Communication Technologies (ICT) at one college in an 
Australian university. 

In total, about 71,240 written comments made on a subject feedback survey by College of Arts 
students in Semesters 1 and 2 in 2007, and Semester 1 in 2008 were analysed using the qualitative 
analysis tool CEQuery (Scott, 2006). A more detailed analysis of 1,866 specific comments on ICT- 
enabled learning was then undertaken. Some 26 different types and uses of ICT-enabled learning were 
identified by students in their comments as a ‘best aspect’ of their university experience. The preferred 
uses of ICT-enabled learning are consistently focused on active learning — online search and use of 
the online library; various forms of peer-supported learning, online discussions, quizzes, tutorials, 
experiment; and receiving prompt and constructive feedback on their learning. The article argues that 
ICT-enabled learning must always be seen as being just one (albeit very important) element among the 
many ‘best aspect’ methods identified by students as engaging them in productive learning. 
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Since the 1990s Australian universities and colleges have been faced with figuring out 
how best to respond to the relentless development of communication and information 
technologies, their rapid influx into our daily lives, the exponential growth in computing 
power, and the rapid growth in internet speeds (Braun, Kadi, & Mahadevan, 2002; Fullan & 
Scott, 2009; Gallagher, 2001; Macnamara, 2007; McLoughlin & Luca, 2001; Segrave & Holt, 
2003; Sharpe, Benfield, & Francis, 2006). The information technology (IT) revolution is 
creating new expectations and opportunities for how university students want to and can 
learn. It questions, for example, whether higher education should remain campus-based or 
become more ‘distributed’, especially given the increased difficulty of travel in large cities 
and the potential for people to work productively for at least part of their week at home. 
Already, traditional universities are no longer viewed as the sole, or even key, repository of 
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leading-edge knowledge or necessarily as being the best place to access it (Fullan & Scott, 
2009). 


The key dimensions of the challenge facing Australian higher education in 
determining how best to deploy ICT-enabled learning appear similar to those found in U.S., 
U.K. and Canadian studies. As reported by Twigg (2003), some U.S. institutions have simply 
bolted new technologies onto existing systems and ‘traditional’ notions of classroom 
instruction, thereby generating additional expenses rather than clear educational, operational 
or financial benefits. Sharpe and Benfield (2005) and Kirkwood and Price (2006) from the 
UK suggest that the majority of e-leaming initiatives in higher education to date have 
reflected a teacher-centred rather than student-focused approach to learning design. Kanuka 
and Kelland from Canada (in press) advise that higher education leaders and teaching 
practitioners should remain cautious regarding their expectations for e-leaming technologies, 
until it can be determined exactly what IT-enabled strategies add to the quality of learning in 
specific discipline areas, in different contexts and with different types of students. 

What the collapse of a range of ventures aimed at setting up solely online learning 
institutions confirms is that ICT-enabled learning strategies must always remain just one 
(albeit very important) element among more than 60 preferred learning methods identified by 
university students (Scott, 2006) that need to be combined and delivered appropriately to 
optimise student learning and retention. 

Admittedly, there are small numbers of students who are quite happy to work totally 
alone and online but, as has been shown (Scott, 2006, 2009), learning is a profoundly social 
experience. We know that students like to learn by doing, using a wide range of practice- 
oriented methods, via active learning in groups, and to be able to contact their tutor for ‘just- 
in-time’ and ‘just-for-me’ assistance. The wide range of ICT tools now available can help 
meet some, but not all of these key engagement and retention elements. For example, they 
have significant potential to allow students to ‘learn in their own time’, to access quality 
assured materials online and download them, and to rapidly search large databases, along with 
a range of active learning options including simulations and social networking. 

Students’ uncertainty about how to use ICT for learning (as distinct from using it for 
social purposes) and their preconceived expectations that university learning will replicate 
schooling create additional challenges (Coates & Rosicka, 2006; Kennedy, Judd, Churchward, 
Gray, & Krause, 2008; Zimitat, 2004). The situation is compounded further by the fact that 
research undertaken in one university (Barraket & Scott, 2001) found that there may still be a 
digital divide in universities, segmented primarily along the lines of social class. This research 
found that people who are first in their family to attend university and whose family incomes 
were low were much less likely to be ‘ICT savvy’ or to have family or friendship networks to 
help them set up and use ICT-enabled learning efficiently. Mori (2008) has identified a 
similar pattern in the UK. 

A number of studies to date have sought to determine the ‘effectiveness’ of different 
uses of ICT introduced by lecturers or universities. However, the results of these studies are 
equivocal because the dependent variables in such studies (performance on tests, exams, etc.) 
often have not been validated as being of a university standard or comparable. In addition, it 
is very hard to separate out the effect of an ICT-enabled learning strategy from the multiple 
influences on student learning. Such issues are explored in more detail in a commissioned 
research and analysis report to the 2008 Review of Australian Higher Education (Scott, 2009). 
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Rather than look at correlations between a range of specific ICT-enabled learning 
methods and learning outcomes, some recent studies have sought to access the higher 
education student experience of using ICT at university more directly, to work with learners 
as they are using ICT in their studies. A good example of this approach is the in-depth 
research undertaken by Creanor, Trinder, Gowan and Howells (2006) with 55 learners. The 
researchers found that email was by far the most used technology (69% reported usage), 
followed by computer-based course materials (47%), computer-based assessments (38%), 
video and audio files (27%), electronic whiteboard (25.5%), and online discussion board 
(22%). Less experience with videoconferencing and learning on a mobile device were 
reported. The researchers conclude that ‘the internet is the first port of call for information, 
with libraries and books taking second place’ (p. 26). In their review of studies on the student 
experience of e-learning in U.K. higher education Sharpe and Benfield (2005) come to similar 
conclusions. The findings in a U.K. study by Mori (2008) suggest that the most useful 
applications of ICT were (in rank order): course-specific materials online, general course 
information online, contacting tutors or lecturers by email or text, online library resources, 
non-digital resources, university’s portal, online submission of work, search of scholarly 
websites, social networking sites to discuss coursework, and participation in online 
communities. Respondents reported varying levels of confidence in using such applications. 
They reported greater comfort in using instant messaging, emails, online discussion, learning 

TM 

management systems like Web CT, Blackboard and Moodle, and accessing course 
materials; and less comfort in using podcasts, making wikis, submitting assignments online 
and using social networking sites. 

In an action research project that focused on identifying the optimum ways to use 
internet-based systems to develop higher levels of learning Kanuka (2005) found that the 
most productive approach involved the use of role-play and case studies. This, stated the 
researcher, was because they fostered collaborative work in a structured fashion and made it 
possible for students to bring in and apply multiple perspectives to the problems addressed. 
This is consistent with approaches known to foster ‘deep learning’ and also the case-based 
based methods found by Sullivan and Rosin (2008) to facilitate the development of ‘practical 
reason’. 

Whereas most research on students’ experience of university, including their 
experience with e-learning, utilises predominantly quantitative data gathered via closed-ended 
questions, this article is focused on accessing the student perspective via qualitative data — 
students’ own written comments on the issue. These types of data are proving to be a lot 
richer than usually expected. Current studies analysing student and graduate comments 
(Bolden & Moscarola, 2000; Kabanoff, Richardson, & Brown, 2003; Scott, 2006) indicate 
that there are important areas of the university experience untapped by existing quantitative 
approaches. Thus, what this article argues is that, in order to determine how best to use ICT as 
part of a broader learning design, it is important to give far more attention to what students 
say in their own words and to incorporate such feedback into universities’ strategic 
development of the area (Poindexter, 2006; Symons, 2006a, 2006b). 

Method 


CEQuery 

In this context, since 2006 all student surveys at one large Australian metropolitan 
multi-campus university — from surveys of the total student experience to those focused on 
specific courses and units of study — invite respondents to identify the best aspects (BA) of 
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their experience and those which most need improvement (NI). The written comments are 
automatically classified by the CEQuery qualitative analysis tool. 

CEQuery is a software tool that facilitates analysis of the written comments on the 
CEQ and any other student feedback survey upon which respondents make open-ended BA 
and NI comments (Scott, 2006). CEQuery allocates comments to 26 subdomains, which are 
grouped into five broad domains — Outcomes, Staff, Support, Assessment and Course 
Design using a custom-tailored dictionary (Attachment 1). The CEQuery dictionary can be 
adjusted by users if they are not satisfied with the accuracy of coding. 

The total number of ‘hits’ (BA + NI) for each domain and subdomain is taken to be a 
proxy measure of perceived importance. This is because it is assumed that, if students choose 
to write (positively or negatively) about an aspect of their university experience in an open- 
ended comment it must be of importance to them. It is also assumed that the odds of students 
making a BA comment (BA/NI) is a proxy measure of perceived quality. For example, when 
the ratio of BA to NI comments in a subdomain is 2.0 it means that there are twice as many 
‘best aspect’ comments as ‘needs improvement’ comments. When the ratio is 0.5, it means 
that there are half as many ‘best aspect’ comments as there are ‘needs improvement’ 
comments. A large number of Australian universities are now using CEQuery and these 
assumptions have been generally endorsed at workshops conducted with them, as well as at a 
wide range of national and international conferences. 

In an earlier study of 280,000 BA and NI comments from 92,000 students in 14 
Australian universities we found comparatively little mention of ICT-enabled learning 
methods and resources and, when they were discussed, the balance between BA and NI 
comments was comparatively even, indicating patchy deployment. We also found that, 
consistent with earlier research, ICT-enabled learning always works best as part of a broader 
learning design and set of university experiences (Scott, 2006; Scott & Alexander, 2000). 
Since the 2006 study the use of and investment in various forms of ICT-enabled learning has 
grown rapidly, with a parallel increase in attention to it in students’ BA and NI comments on 
learning methods. At the same time an increasing variety of uses is being reported by 
students. 

With this in mind, this article reports on an analysis of more than 76,000 BA and NI 
comments in a survey of student feedback on units undertaken in one university over the 
period 2007-2008. The article first identifies the overall patterns of what students give most 
attention to in their comments and indicates for each CEQuery domain and subdomain the 
balance of BA and NI comments. Then it looks specifically at the learning methods 
subdomain and identifies the different clusters of learning methods students refer to as a best 
aspect of their university experience — of which more than 60 particular strategies have been 
cited. Finally, it hones in on what students identify as the best aspects of their experience with 
the methods specifically involving ICT-enabled learning. 

Participants 

The survey that generated the data for the study evaluates individual subjects each 
time they are offered. It is sent to all of the university’s currently enrolled students each year. 1 
The response rate ranges from 50% to 55% and the response samples are generally 
representative of the university’s profile in terms of a wide range of demographic variables. 
The proportions of survey respondents and total student population do not vary considerably 
by gender (difference range = 7.6%—9.2%), college (0.8%-4.0%), campus (0.2%-4.0%), level 
of study (1.9%-9.0%), Aboriginal or Torres Strait Islander descent (0.0%-0.1%), students 
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speaking a language other than English at home (9.3%-10.6%), students with a disability 
(2.7%-3.0%), and international students (1.2%-1.6%). Some 71,240 BA and NI comments 
were made on the survey by students in the Humanities and Social Sciences field of education 
and in related areas over semesters 1 and 2 in 2007 and semester 1 in 2008. 

Data analysis and results 


CEQuery analysis of the 71,240 comments 

Attachment 2 provides the overall results. Of the 71,240 comments made by students 
31,983 were made about various aspects of course design (flexibility — 4,749; learning 
methods — 15,443; practical-theory links — 2,017; relevance — 3,718; and structure — 
4,444). Some comments (e.g., ‘it was good’, ‘yes’, ‘don’t know’, etc.) could not be allocated 
to any CEQuery subdomain and are classified as ‘unspecified’. 

The learning methods subdomain attracted the highest number of BA and NI 
comments of all 26 subdomains that make up the CEQuery analytical categories. Attachment 
3 gives the full range of learning methods identified in our CEQuery studies in 14 universities 
across Australia (Scott, 2006). The range identified in the present study is similar. 

The 15,443 comments on learning methods made by students were made up of 10,214 
BA comments and 5,229 NI comments giving the odds of a BA comment for this subdomain 
of 2 to 1. Attachment 4 provides more specific data sorted by learning methods category. 

A study of the 10,214 BA comments for learning methods was then undertaken. This 
revealed that, in this field of education: 

• 8,119 of the BA comments were about various face-to-face methods with the odds of a 
BA comment for this area being 1.9 to 1 

• 790 were about various forms of ICT enabled learning (odds of a BA comment 7 in 10) 

• 569 were about independent learning methods (odds of a BA comment 6 in 10) 

• 1333 were about practical and real world learning (odds of a BA comment 2.6 to one) 

• 165 were about the use of simulations and lab-based learning (odds of a BA comment 2 
to 1). 

The BA results for ICT-enabled learning 

An analysis of the 790 BA comments for ICT-enabled learning was then undertaken. 

It revealed that some 26 different types and uses of ICT-enabled learning were identified by 
students as a ‘best aspect’ in their comments. A study of the 1,076 NI comments on the ICT- 
enabled learning methods area revealed that the large majority were simply calling for more 
systematic application or more effective delivery of the uses identified as a best aspect. Table 
1 shows the specific uses of ICT-enabled learning identified as a BA, with the number of 
times each was mentioned. 
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Table 1 

Best Aspects of ICT-enabled Learning by Frequency of Occurrence in Student Comments 


_ Aspect of ICT-enabled learning _ Count 

WebCT as a convenient, one stop access point to resources 120 

Provision of online lectures 2 117 

Online discussion forum/discussion board 97 

A range of active, online learning activities, e.g., simulations 80 

Online tutor/tutorials 78 

Use of ICT-enabled learning to support a more flexible time table 3 68 

Online quiz 55 

Use of images for analysis 31 

Viewing a DVD 26 

Online assessment/web-based assignments 19 

Online assignment submission 14 

Photography, including use of photoshop 14 

Online help/support 10 

Blog 10 

Lectures provided on CD 10 

Online audio files/sound clips etc 9 

Designing and creating a web-site 8 

Being able to contact & get feedback from lecturer via email 8 

Use of the net for peer feedback 4 

Online planning of group work 3 

Online debate 2 

Online learning games 2 

Online portfolio 2 

Online video 1 

Online experiment 1 

IT practical 1 

Total 790 


Discussion and conclusions 

As Table 1 shows, ICT-enabled learning has a wide range of BA uses but, as 
Attachment 3 demonstrates, it must always be seen as being just one component of the much 
broader set of active, practice-oriented and integrated strategies that engage students in 
productive learning. 

The BA uses of ICT-enabled learning identified in Table 1 are consistently concerned 
with active learning — online search, peer-supported learning, experiment, receipt of 
feedback and so on. Equally, the preferred uses are those that make learning as convenient 
and cost-productive as possible. What does not attract support is the passive delivery of large 
amounts of content unrelated to assessment content or material that is not directly and 
meaningfully located within a broader learning structure. Attachment 2 indicates that course 
structure and integration continue to be a key area for overall quality improvement with the 
current odds of a BA comment in this and other datasets being 3 in 10. 

The analysis also reveals some support for the use of more flexible learning designs — 
including considerable (but not total) support for the use of alternative weeks of face-to-face 
learning complemented by ICT-enabled learning via WebCT or an equivalent system when 
students are not in attendance. 

There is considerable alignment between comments on this specific area and the 
broader findings from CEQuery — that what engages students in productive learning and 
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optimises their retention is the total experience of the university — peer support, ‘just-in- 
time’ and ‘just-for-me’ assistance; the ability for students to ‘learn in their own time’; active 
problem-based learning; easy access to online learning resources and so on — not just what 
happens in the ‘traditional’ classroom. 

The results also show that ICT-enabled learning involves the use of far more than the 
internet. There are also some suggestions in the data that not all students have the broadband 
to enable them to ‘learn in their own time’ at home, suggesting that the digital divide may, as 
our 2000 study (Barraket & Scott, 2001) found, still exist. A key predictor in this regard is 
coming from a low income background and being the first in one’s family to attend 
university. 

One more important message from this analysis is that CEQuery can be a valuable 
complement to the other tracking and improvement systems already being used in 
universities. At the university that has been the focus of this study a specific breakdown of 
comments in all CEQuery domains and subdomains is undertaken not only for each college, 
but also for each school. This enables the colleges to identify subdomains in CEQuery 
attracting high odds of a BA comment in one unit and to determine what is being done for 
possible dissemination to similar units elsewhere where the odds of a BA comment in the 
same subdomain are lower. It also enables each college to ascertain potentially relevant 
solutions to key improvement priorities identified in the qualitative data from a range of 
feedback surveys. Because CEQuery enables the user to look at all of the comments for a 
particular subdomain in one file, insights into what students had in mind when allocating their 
ratings for quantitative items can be rapidly gained. 

This article identifies the potential to access the student voice more consistently when 
seeking to develop effective uses of IT-enabled learning as part of a broader learning system. 
It confirms that it is the total experience of the university that shapes students’ judgments of 
quality, engages them in productive learning and retains them, but that IT-enabled learning 
certainly has an important role to play. It indicates that the general rules for effective learning 
— that it should be active, not passive, ‘just in time’ and ‘just for me’, that it should link 
theory with practice and be collaborative — apply specifically to best aspect uses of ICT 
identified in studies like the present one. 

The analysis of what students say in their own words about the best aspects of their 
learning using the CEQuery tool is of increasing interest to higher educators. Its use in the 
present study to first explore ‘best aspect’ learning methods in general, and then those 
identified by students as being productive in the specific subdomain of IT-enabled learning, is 
very much a first step. Its further use as part of a coordinated project across all of Australia’s 
universities and not just in this field of education but others is commended. 

Endnotes 

1. The survey comprises 13 items relating to the quality of individual subjects, including their content and 
relevance, assessment, learning environment, learning experience and outcomes, and workload. Each 
item is ranked by students on a five-point Likert scale where 1 represents strong disagreement and 5 - 
strong agreement. At the end of the survey respondents are asked to provide comments on the best 
aspects (BA) of the University’s performance, and on those most needing improvement (NI). 

2. The preference here was to have the lecture slides, notes in advance of a formal lecture. Other options 
cited included the use of videotaped lectures online. Less mention was made of audiotaped lectures or 
podcasts as a BA method. 
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3. A large number of these comments spoke favourably of the use in some units of a flexible timetable in 

which one week involved face-to-face activities and the next used various forms of ICT-enabled and 
online learning. Note, however, there was also a considerable number of ‘needs improvement’ 
comments on this issue, indicating that implementation is currently patchy. 
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CEQuery Domains and Subdomains 

Attachment 1 


Domain 

Subdomain 

Outcomes 


Intellectual 

Work application/career 

Further learning 

Personal 

Interpersonal 

Knowledge/skills 

Staff 


Accessibility and responsiveness 

Teaching skills 

Practical experience (current) 

Quality and attitude 

Course Design 


Practical-theory links 

Relevance (to work/life/discipline) 
Flexibility/responsiveness 

Methods of learning and teaching 

Structure and expectations 

Assessment 


Relevance 

Marking 

Expectations 

Feedback/Return 

Standards 

Support 


Library 

Learning resources 

Infrastructure/environment 

Student administration 

Student services 

Social affinity/support 


CEQuery Subdomains: Specific Definitions 


OUTCOMES 

Intellectual 

Development of analytical skills, critical thinking, creativity, problem-solving, diagnostic abilities; ability to 
‘see the key issue’ in a welter of information, come to a justified decision in a tricky situation, trace out the 
consequences of various options for action, understand one’s key assumptions, see ‘the big picture’ and 
‘think on one’s feet’. Intellectual capabilities interact with Personal and Interpersonal ones. 

Work application/career 

Includes gaining promotion, improved employability, improved workplace performance, direct application of 
what was learnt at work. 

Further learning 

Going on to further and higher study as a result of the course; commitment to life-long learning. In the case 
of NI comments students may talk more about the blocks they experienced or the reasons why the course 
didn’t motivate them to go on to further study. 
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Personal 

All aspects of personal Emotional Intelligence identified in recent studies of successful graduates and other 
research (see Vescio, 2005) for example, the ability to remain calm when things go wrong, self-confidence, 
sense of ‘efficacy’, willingness to take negative feedback, ability to tolerate ambiguity, persevere and 
maintain self-motivation, independence, self understanding etc). Also includes comments about the personal 
satisfaction that comes from completing a higher-education program. 

Interpersonal 

This covers not just written and verbal communication skills but key aspects of social Emotional Intelligence 
identified in the successful graduate studies (e.g., the ability to work with a wide diversity of people, a 
developed understanding of cultural differences, an ability to work productively as part of a team, 
development and use of peer/other networks). See Scott and Yates (2002), Vescio (2005) for more detail on 
these concepts. NI comments tend to talk about the blocks in communication during the course that prevented 
the development of the desired interpersonal outcomes — staff and students with poor communication skills 
in English are regularly cited in this context. 

Knowledge/skills 

Includes both generic skills/knowledge (e.g., the ability to chair a meeting, use computers; self-teaching 
skills, library search skills, information literacy and skills of observation) and profession/discipline-specific 
skills/knowledge (e.g., knowledge of a particular statute in law, or specific skills for use in a laboratory, etc). 
Also includes research skills. 

STAFF 

Accessibility and responsiveness 

Ability to contact staff (face-to-face, online, by telephone etc), staff availability, how and when they respond, 
their willingness to support students, as well as comments about the interface between staff- student ratios 
and staff accessibility and responsiveness. 

Teaching skills 

Staff ability to teach and convey knowledge; their effectiveness, creativity, organisation and enthusiasm as 
lecturers as distinct from comments on how knowledgeable they are, or how they behave outside the 
classroom. 

Practical experience (current) 

How up-to-date, ‘in touch’ and linked staff are with current professional or disciplinary practice through, for 
example, being a current practitioner. Extent to which there is use of guest lecturers; staff ability to use ‘real 
world’ anecdotes to make their teaching more relevant. 

Quality and attitude 

Staff members’ ability to inspire; their enthusiasm, promptness in coming to class, reliability, levels of 
organisation, engagement; their professionalism, organisation, commitment to the area taught, interpersonal 
skills and clarity of communication including English-language skills. 

COURSE DESIGN 

Practical-theory links 

The consistency with which a course seeks to link and balance theory with practice, designs in a range of 
practice-oriented experiences directly connects to related theory. The extent to which it is professionally 
oriented and applied in its design. 

Relevance (to work/life/discipline) 

How interesting, engaging, current, and relevant course content is. Also includes comments about courses 
being personally relevant to the key interests and meeting students’ other needs. 

Flexibility/responsiveness 

This includes comments on the extent to which the course design provides flexible/responsive learning paths 
(electives/majors/submajors); choice; negotiated learning; flexible attendance patterns; flexible delivery; ease 
of access to learning and assistance to determine which path is best. This subdomain has links to course 



Journal of Institutional Research, 14(2), 1 - 16 . 


12 


design but here the focus is on the extent to which the course is able to respond to the particular backgrounds, 
abilities, needs and experiences of students as opposed to having a single ‘one size fits all’ model. 

Methods of learning and teaching 

Approximately 60 different learning and teaching methods have been identified including: lectures, group 
work, seminars, tutorials, specific practical, real-life learning methods (practicum, internships, coop ed., 
moots, simulations, work placements, field trips, clinical placements, industry and practical legal training, 
etc); use of prior learning of students; camps; lab-work to learning contracts, site visits, experiments, various 
forms of IT-enabled learning, simulations, teleconferences, guest speakers, specific peer/team learning 
methods and case-study analysis. Appropriate use of interactive learning methods is a recurring theme in 
students’ BA comments. 

Structure and expectations 

Structure: subject balance and distinctiveness from each other, subject quality, overall load and amount of 
content to be learnt, appropriate sequence of learning, overlap between subjects, prerequisites, admission 
levels, timetable, overview of field, recognition of prior learning (RPL), the appropriateness of the modes of 
learning used fpt/ft, mixed mode, multi-site, intensive, work-based, distance, online etc.). Also includes 
comments about the appropriateness, timing, length and variety of mix of learning methods used, the extent 
to which the course has depth, a clear direction, is integrated, and has an overall integrity. 

Expectations: management and clarity of information provided, course rules, access to staff, resources, 
university processes. Also includes comments about alignment between course prospectus and delivery and 
actual availability of advertised electives. 

ASSESSMENT 

Relevance 

Extent to which assessment tasks are perceived to be real-world, applied, up-to-date, integrated, relevant to 
current and future professional or disciplinary practice and focused on ‘real world’ problems. Also covers 
comments where students discuss the extent to which assessment is interesting, challenging, engaging, 
appropriate and how well it matches what was taught and the stated subject/course objectives. 

Marking 

Consistency and reliability of marking; fair assessment of group work projects and NESB student work. 
Covers reliability across different assessment methods: short answer; online; practice-based; group-based etc. 
Also includes extent to which plagiarism and cheating are detected, comments about ‘soft-marking’ and the 
confusion between norm-referenced and criterion-referenced assessment in determining grades. 

Expectations 

Provision of clear assessment tasks and expectations on how to tackle and present them; clear submission 
deadlines, guidelines rules and grading criteria. Provision of examples of work, to give an operational picture 
of different grades and quality of work in each subject. 

F eedbaek/return 

Promptness with which assignments are returned, use of staged deadlines, quality of the feedback received 
including the extent to which markers comment on what was done well, explicitly identify key areas for 
improvement and say how improvements could have been achieved — with specific attention to the grading 
criteria distributed at the start of the subject. 

Standards 

Assessment which is at a university standard, which requires higher-order thinking more than rote 
memorisation from text books; is interesting, and negotiated; assessment that is valid (i.e., demonstrably 
focuses on the key capabilities that graduates will need to succeed in the first years of work in a specific 
profession or discipline). Includes comments about rote learning, industry recognition, over-assessment, 
range and appropriateness of assessment methods used, assessment load, plagiarism management, appeals, 
extensions, alignment between what is taught and tested, prerequisites, norm versus criterion-referenced 
assessment, submission and security, timing, weighting, and consistency of assessment quality and demands 
between subjects and courses at the same level. 
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SUPPORT 

Library 

Library collections, services, ease of access, facilities, equipment, efficiency, online services as well as face- 
to-face services, borrowing services and rules, fines. 

Learning resources 

Quality and availability of textbooks, print & digital support materials, course outlines, study guides, lecture 
notes, course readings, online learning resources, self-teaching materials, CD-ROMs, video, TV, 
photographic and sound resources. 

Infrastructure/environment 

Classroom and lab quality, class sizes and levels of crowding, quality of computers and technical 
infrastructure, equipment levels and quality, ease of access to physical facilities and their quality, campus 
environment, equipment levels, social spaces. Also comments about funding levels for facilities and financial 
support at universities. 

Student administration 

Enrolment systems (online and offline), exam scheduling, fees processes, administrative advice, exemptions, 
graduation processes, delivery of transcripts, accuracy of fees’ invoices, grievance processes, results, 
scholarships, admission, admin staff responsiveness, timetabling. Includes ease of access to student 
administration services and the extent to which queries and problems are followed up promptly and resolved. 
Also includes comments about efficiency, levels of bureaucracy. 

Student services 

Learning support services (English for academic purposes, study assistance, information literacy, transition to 
university programs, orientation etc), careers. Services to DEST-defined equity groups including ATSI and 
NESB students, along with counselling services. Comments about the helpfulness of support service staff 
including IT-enabled learning support. Both IT-enabled and face-to-face. 

Social affinity/support 

Comments that relate to the sense of ‘belonging’ that comes from a welcoming, friendly, approachable 
environment and culture and set of relationships among both staff and students. Comments which indicate 
that the student feels s/he is seen not as a number but an individual. Comments about levels of engagement or 
isolation felt by students. Also covers comments on the wide range of formal and informal types of social 
support, in particular peer support but also a general culture of support and service, ability to network, 
interaction with others, the development and use of reciprocal relationships. For interactions with staff it 
includes the presence of a ‘service-oriented’ culture. 



Attachment 2 


CEQuery Analysis by School, 2007.1-2008.1 


CEQuery Analysis by School, 2007.1-2008.1 

Domain / 

Contemporary Arts 

Education 

Humanities & Law 

Psychology 

Social Sciences 

College of Arts 

Sub Domain 

BA 

Nl 

Emu 

EE3=E3 

BA 

Nl 

EQED 


BA 

Nl 

Ezznn 


BA 

Nl 

Ezznn 

EMM 

BA 

Nl 

EBED 

Emm 

BA 

Nl 

EHED 

Emm 

assessment 

529 

1,137 

1,666 


463 

2,115 

2,578 


581 

1,417 

1,998 

■ 

425 

Mwnm 

1,680 


571 

1,439 




7,363 



expectations 


411 

450 

0.1 

91 

824 

915 

0.1 

86 

423 

509 


67 


451 

0.2 

87 

515 




2,557 


0.1 

feedback 


170 

223 

0.3 

47 

317 

364 

0.1 

67 

239 

306 


27 



0.1 

49 

223 




1,161 

1,404 

0.2 

marking 

18 

91 

109 

0.2 

21 

211 

232 

0.1 

51 

133 

184 

0.4 

21 

114 


0.2 

30 

160 

190 

0.2 

141 

709 

850 

0.2 

relevance 

155 

59 

214 

2.6 

107 

107 

214 

1.0 

108 

104 

212 

1.0 

110 

89 

199 

1.2 

147 

86 

233 

1.7 

627 

445 

1,072 

1.4 

standards 

193 

343 


0.6 

153 

600 

753 

0.3 

185 

440 

625 

0.4 

126 

398 

524 

0.3 

178 

383 

561 

0.5 

835 

2,164 

2,999 

0.4 

unspecified 

71 

63 


1.1 

44 

56 

100 

0.8 

84 

78 

162 

1.1 

74 

58 

132 

1.3 

80 

72 

152 

1.1 

353 

327 

680 

1.1 

course_design 

2,915 

1,724 

4,639 


4,064 

3,145 

7,209 


4,860 

MMiM 

■*I=M 


3,334 


5,435 


3,856 

2,584 

6,440 


19,029 

12,954 

31,983 


flexibility 

470 

212 

682 

2.2 

621 

457 

1,078 

1.4 

740 

419 

1,159 

1.8 

635 

306 

941 

2.1 

551 

338 

889 

1.6 

3,017 


4,749 

1.7 

methods 

1,588 

668 

2,256 

2.4 

2,131 

1,359 

3,490 

1.6 

2,670 

1,345 

4,015 

2.0 

1,645 

806 

2,451 

2.0 

2,180 

1,051 

3,231 

2.1 

10,214 


15,443 

2.0 

practical theory links 

390 

83 

473 

4.7 

541 

163 

704 

3.3 

119 

79 

198 

1.5 

244 

81 

325 

3.0 


78 

317 

3.1 

1,533 

484 


3.2 

relevance 

232 

224 

456 

1.0 

477 

297 

774 

1.6 

641 

443 

1,084 

1.4 

446 

194 

640 

2.3 


301 

764 

1.5 

2,259 

1,459 

EBEI 

1.5 

structure 

133 

450 

583 

0.3 

137 

807 

944 

0.2 

362 

956 

1,318 

0.4 

150 

608 

758 

0.2 

179 

662 

841 

0.3 

961 

3,483 

4,444 

0.3 

unspecified 

102 

87 

189 

1.2 

157 

62 

219 

2.5 

328 

158 

486 

2.1 

214 

106 

320 

2.0 

244 

154 

398 

1.6 

1,045 

567 

1,612 

1.8 

outcomes 

830 


968 


980 


1,172 


1,199 

233 

1,432 


417 

110 

527 


854 

135 

989 


4,280 

808 



further learning 

0 

H 

0 

n.a 

8 

mm 

8 

n.a 

13 

0 

13 

n.a 

1 

0 

1 

n.a 

8 

0 

8 

n.a 

30 

0 

30 

n.a 

intellectual 

305 

11 

316 

27.7 

324 

ii 

335 

29.5 

395 

20 

415 

19.8 

161 

5 

166 

32.2 

330 

5 

335 

66.0 

1,515 

52 

1,567 

29.1 

interpersonal 

67 

9 

76 

7.4 

21 

7 

28 

3.0 

119 

17 

136 

7.0 

14 

9 

23 

1.6 

41 

7 

48 

5.9 

262 

49 

311 

5.3 

knowledge / skills 

291 

91 

382 

3.2 

325 

95 

420 

3.4 

541 

165 

706 

3.3 

135 

74 

209 

1.8 

313 

90 

403 

3.5 

1,605 

515 

2,120 

3.1 

personal 

34 

0 

34 

n.a 

47 

1 

48 

47.0 

28 

1 


28.0 


0 

28 

n.a 

31 

1 

32 

31.0 

168 

3 

171 

56.0 

unspecified 

38 

6 

44 

6.3 

42 

8 

50 

5.3 

45 

8 


5.6 


2 

25 

11.5 

43 

10 

53 

4.3 

191 

34 

225 

5.6 

work application 

95 

21 

116 

4.5 

213 

70 

283 

3.0 

58 

22 

80 

2.6 

55 

20 

75 

2.8 

88 

22 

110 

4.0 

509 

155 

664 

3.3 

staff 

1,168 

494 

1,662 


2,340 

903 



3,267 

729 

3,996 


1,363 

341 

1,704 


2,204 

685 

2,889 


10,342 

3,152 

13,494 


accessibility 

366 

189 

555 

1.9 

617 

283 


2.2 

652 

189 

841 

3.4 

297 

92 

389 

3.2 

469 

207 

676 

2.3 

2,401 

960 

^1 

2.5 

practical experience 

15 

4 

19 

3.8 

20 

4 


5.0 

6 

2 

8 

3.0 

12 

0 

12 

n.a 

13 

1 

14 

13.0 

66 

11 

mm 

6.0 

quality 

508 

132 

640 

3.8 

1,078 

308 


3.5 

1,622 

260 

1,882 

6.2 

640 


756 

5.5 

1,071 

216 

1,287 

5.0 


1,032 


4.8 

teaching skills 

231 

163 

394 

1.4 

582 

294 

876 

2.0 

860 

266 

1,126 

3.2 

352 


479 

2.8 

568 

238 

806 



1,088 

3,681 

2.4 

unspecified 

48 

6 

54 

8.0 

43 

14 

57 

3.1 

127 

12 

139 

10.6 

62 


68 

10.3 

83 

23 

106 


363 

61 

424 

6.0 

support 

652 

806 

1,458 


974 

996 

MEEiM 


2,029 

2,092 

KSE9 


500 

798 

HEU 


715 

■Em 

1,896 


ESQ! 

5,873 

ll'HH 


infrastructure 

197 

277 

474 

0.7 

261 

273 

534 

1.0 

54 

205 

259 

0.3 

18 

132 

150 

0.1 

64 

219 

283 

0.3 

594 

1,106 

1,700 

0.5 

learning resources 

304 

389 

693 

0.8 

411 

465 

876 

0.9 

1,628 

1,601 

3,229 

1.0 

348 

535 

883 

0.7 

411 

783 

1,194 

0.5 

3,102 

3,773 

6,875 

0.8 

library 

3 

13 

16 

0.2 

1 

11 

12 

0.1 

10 

44 

54 

0.2 

5 

18 

23 

0.3 

4 

29 

33 

0.1 

23 

115 

138 

0.2 

social affinity 

137 

84 

221 

1.6 

265 

181 

446 

1.5 

306 

202 

508 

1.5 

107 

87 

194 

1.2 

205 

105 

310 

2.0 

1,020 

659 

1,679 

1.5 

student administration 

2 

25 

27 

0.1 

10 

52 

62 

0.2 

14 

37 

51 

0.4 

14 

23 

37 

0.6 

4 

40 

44 

0.1 

44 

177 

221 

0.2 

student services 

5 

3 

8 

1.7 

10 

11 

21 

0.9 

4 

3 

7 

1.3 

2 

3 

5 

0.7 

11 

2 

13 

5.5 

32 

22 

54 

1.5 
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Attachment 3 


CEQuery ‘Best Aspect’ Learning Methods Sorted by Type 


FACE-TO-FACE 

PRACTICE-ORIENTED & 
REAL WORLD’ 

INDEPENDENT STUDY 

SIMULATIONS & 
LABS 

ICT- SUPPORTED 
LEARNING METHODS 

• Lecture (interactive) 

• Group project small group 
work 

• Tutorial 

• Class-work exercises 

• Discussion, sharing ideas 

• Seminar/ individual 
presentation 

• Workshop 

• Debate 

• 1:1 consultation 

• Mentor (peer or staff) 

• Conference/symposium 

• Forum/panel 

• Exhibition 

• Peer learning & support 

• Group dynamics exercises 

• Critique of student 

• Production/creation 

• Buzz group 

• Clinical placement 

• Practicum / practical teaching 

• Teaching ‘rounds’ 

• Practical legal training 

• Cooperative Education 

• Work experience or placement for 
work-based learning, 

• Professional mentor 

• “Learning by doing” 

• Field study/work/trip/site visit 

• Camps 

• Addressing real-life problems 

• Use of guest speakers, 
industry/prof, representatives 

• Practical work at university 

• Design Studio 

• Artistic production 

• Placement or study overseas, or in 
another Australian university 

• Real life case study 

• Applying learning to work 
problems 

• Learning by completing 
assignments /essays 

• Writing a research or 
community service report 

• Use of self¬ 
teaching/distance education 
packages 

• Self-teaching guide 

• Project report writing 

• Proposal writing 

• Learning contract 

• Mock trial 

• Role play 

• Simulated 
interview 

• Hypothetical 

• Educational game 

• Discovery learning 

• Experiments 

• Lab work 

• In tray exercises 

• Use of a simulator 

• Online search for information/web 
sites 

• Web-based learning e.g.WebCT 

• Blogs, My Space, etc. 

• On-line study 

• Email contact with staff/students 

• SMS with staff/students 

• Individual phone contact with 
Staff/students 

• Teleconference 

• Tele-tutorial 

• Video conference 

• Learning using 

Podcasts, MP3 

Radio 

Audio Tapes, CDs 

TV 

Video/DVD 

Photos, slides, 

Digital images 
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Attachment 4 


Learning Methods by School, 2007.1-2008.1 


Learning 

Contemporary Arts 

Education 

Humanities & Languages 

Method 

BA 

Nl 

BA + Nl 

Odds BA 

BA 

Nl 

BA + Nl 

Odds BA 

BA 

Nl 

BA + Nl 

Odds BA 

face-to-face 

1,060 

488 

1,548 

2.2 

1,422 

891 

2,313 

1.6 

2,425 

1,232 

3,657 

2.0 

ICT supported Irng methods 

119 

115 

234 

1.0 

239 

495 

734 

0.5 

195 

225 

420 

0.9 

independent study 

87 

106 

193 

0.8 

169 

395 

564 

0.4 

117 

163 

280 

0.7 

prac oriented/real world 

233 

63 

296 

3.7 

418 

171 

589 

2.4 

137 

59 

196 

2.3 

simulation & labs 

42 

22 

64 

1.9 

22 

14 

36 

1.6 

9 

11 

20 

0.8 


Learning 

Psychology 

Social Sciences 

College of Arts 

Method 

BA 

Nl 

BA + Nl 

Odds BA 

BA 

Nl 

BA + Nl 

Odds BA 

BA 

Nl 

BA + Nl 

Odds BA 

face-to-face 

1,370 

689 

2,059 

2.0 

1,842 

904 

2,746 

2.0 

8,119 

4,204 

12,323 

1.9 

ICT supported Irng methods 

150 

116 

266 

1.3 

87 

125 

212 

0.7 

790 

1,076 

1,866 

0.7 

independent study 

69 

105 

174 

0.7 

127 

167 

294 

0.8 

569 

936 

1,505 

0.6 

prac oriented/real world 

248 

90 

338 

2.8 

297 

124 

421 

2.4 

1,333 

507 

1,840 

2.6 

simulation & labs 

76 

29 

105 

2.6 

16 

8 

24 

2.0 

165 

84 

249 

2.0 
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